Beyond simple laboratory studies: Developing sophisticated models to study rich behavior.

Psychology and neuroscience are concerned with the study of behavior, of internal cognitive processes, and their neural foundations. However, most laboratory studies use constrained experimental settings that greatly limit the range of behaviors that can be expressed. While focusing on restricted settings ensures methodological control, it risks impoverishing the object of study: by restricting behavior, we might miss key aspects of cognitive and neural functions. In this article, we argue that psychology and neuroscience should increasingly adopt innovative experimental designs, measurement methods, analysis techniques and sophisticated computational models to probe rich, ecologically valid forms of behavior, including social behavior. We discuss the challenges of studying rich forms of behavior as well as the novel opportunities offered by state-of-the-art methodologies and new sensing technologies, and we highlight the importance of developing sophisticated formal models. We exemplify our arguments by reviewing some recent streams of research in psychology, neuroscience and other fields (e.g., sports analytics, ethology and robotics) that have addressed rich forms of behavior in a model-based manner. We hope that these "success cases" will encourage psychologists and neuroscientists to extend their toolbox of techniques with sophisticated behavioral models - and to use them to study rich forms of behavior as well as the cognitive and neural processes that they engage.

Enhanced efficiency in visually guided online motor control for actions redirected towards the body midline.

Reaching objects in a dynamic environment requires fast online corrections that compensate for sudden object shifts or postural changes. Previous studies revealed the key role of visually monitoring the hand-to-target distance throughout action execution. In the current study, we investigate how sensorimotor asymmetries associated with space perception, brain lateralization and biomechanical constraints, affect the efficiency of online corrections. Participants performed reaching actions in virtual reality, where the virtual hand was progressively displaced from the real hand to trigger online corrections, for which it was possible to control the total amount of the redirection and the region of space in which the action unfolded. The efficiency of online corrections and the degree of awareness of the ensuing motor corrections were taken as assessment variables. Results revealed more efficient visuo-motor corrections for actions redirected towards, rather than away from the body midline. The effect is independent on the reaching hand and the hemispace of action, making explanations associated with laterality effects and biomechanical constraints improbable. The result cannot either be accounted for by the visual processing advantage in the straight-ahead region. An explanation may be found in the finer sensorimotor representations characterizing the frontal space proximal to body, where a preference for visual processing has been documented, and where high-value functional actions, like fine manipulative skills, typically take place. This article is part of a discussion meeting issue 'New approaches to 3D vision'.

Interception of virtual throws reveals predictive skills based on the visual processing of throwing kinematics.

Predicting the outcome of observed actions is fundamental for efficient interpersonal interactions. This is evident in interceptive sports, where predicting the future ball trajectory could make apart success and fail. We quantitatively assessed the predictive abilities of non-trained adults intercepting thrown balls in immersive virtual reality. Participants performed better when they could see the complete throwing action in addition to the ball flight, and they were able to move toward the correct direction when the ball flight was occluded. In both cases, performance varies with the individual motor style of the thrower. These results prove that humans can effectively predict the unfolding of complex full-body actions, with no need to extensively practice them, and that such predictions are exploited online to optimize interactive motor performance. This suggests that humans hold a functional knowledge of how actions recurrent in the human motor repertoire map into the changes brought to the environment.

Active inference unifies intentional and conflict-resolution imperatives of motor control.

The field of motor control has long focused on the achievement of external goals through action (e.g., reaching and grasping objects). However, recent studies in conditions of multisensory conflict, such as when a subject experiences the rubber hand illusion or embodies an avatar in virtual reality, reveal the presence of unconscious movements that are not goal-directed, but rather aim at resolving multisensory conflicts; for example, by aligning the position of a person's arm with that of an embodied avatar. This second, conflict-resolution imperative of movement control did not emerge in classical studies of motor adaptation and online corrections, which did not allow movements to reduce the conflicts; and has been largely ignored so far in formal theories. Here, we propose a model of movement control grounded in the theory of active inference that integrates intentional and conflict-resolution imperatives. We present three simulations showing that the active inference model is able to characterize movements guided by the intention to achieve an external goal, by the necessity to resolve multisensory conflict, or both. Furthermore, our simulations reveal a fundamental difference between the (active) inference underlying intentional and conflict-resolution imperatives by showing that it is driven by two different (model and sensory) kinds of prediction errors. Finally, our simulations show that when movement is only guided by conflict resolution, the model incorrectly infers that is velocity is zero, as if it was not moving. This result suggests a novel speculative explanation for the fact that people are unaware of their subtle compensatory movements to avoid multisensory conflict. Furthermore, it can potentially help shed light on deficits of motor awareness that arise in psychopathological conditions.

Active strategies for multisensory conflict suppression in the virtual hand illusion.

The perception of our body in space is flexible and manipulable. The predictive brain hypothesis explains this malleability as a consequence of the interplay between incoming sensory information and our body expectations. However, given the interaction between perception and action, we might also expect that actions would arise due to prediction errors, especially in conflicting situations. Here we describe a computational model, based on the free-energy principle, that forecasts involuntary movements in sensorimotor conflicts. We experimentally confirm those predictions in humans using a virtual reality rubber-hand illusion. Participants generated movements (forces) towards the virtual hand, regardless of its location with respect to the real arm, with little to no forces produced when the virtual hand overlaid their physical hand. The congruency of our model predictions and human observations indicates that the brain-body is generating actions to reduce the prediction error between the expected arm location and the new visual arm. This observed unconscious mechanism is an empirical validation of the perception-action duality in body adaptation to uncertain situations and evidence of the active component of predictive processing.

The road towards understanding embodied decisions.

Most current decision-making research focuses on classical economic scenarios, where choice offers are prespecified and where action dynamics play no role in the decision. However, our brains evolved to deal with different choice situations: "embodied decisions". As examples of embodied decisions, consider a lion that has to decide which gazelle to chase in the savannah or a person who has to select the next stone to jump on when crossing a river. Embodied decision settings raise novel questions, such as how people select from time-varying choice options and how they track the most relevant choice attributes; but they have long remained challenging to study empirically. Here, we summarize recent progress in the study of embodied decisions in sports analytics and experimental psychology. Furthermore, we introduce a formal methodology to identify the relevant dimensions of embodied choices (present and future affordances) and to map them into the attributes of classical economic decisions (probabilities and utilities), hence aligning them. Studying embodied decisions will greatly expand our understanding of what decision-making is.

A Hessian-based decomposition characterizes how performance in complex motor skills depends on individual strategy and variability.

In complex real-life motor skills such as unconstrained throwing, performance depends on how accurate is on average the outcome of noisy, high-dimensional, and redundant actions. What characteristics of the action distribution relate to performance and how different individuals select specific action distributions are key questions in motor control. Previous computational approaches have highlighted that variability along the directions of first order derivatives of the action-to-outcome mapping affects performance the most, that different mean actions may be associated to regions of the actions space with different sensitivity to noise, and that action covariation in addition to noise magnitude matters. However, a method to relate individual high-dimensional action distribution and performance is still missing. Here we introduce a decomposition of performance into a small set of indicators that compactly and directly characterize the key performance-related features of the distribution of high-dimensional redundant actions. Central to the method is the observation that, if performance is quantified as a mean score, the Hessian (second order derivatives) of the action-to-score function determines how the noise of the action distribution affects performance. We can then approximate the mean score as the sum of the score of the mean action and a tolerance-variability index which depends on both Hessian and action covariance. Such index can be expressed as the product of three terms capturing noise magnitude, noise sensitivity, and alignment of the most variable and most noise sensitive directions. We apply this method to the analysis of unconstrained throwing actions by non-expert participants and show that, consistently across four different throwing targets, each participant shows a specific selection of mean action score and tolerance-variability index as well as specific selection of noise magnitude and alignment indicators. Thus, participants with different strategies may display the same performance because they can trade off suboptimal mean action for better tolerance-variability and higher action variability for better alignment with more tolerant directions in action space.

Rethinking GPS navigation: creating cognitive maps through auditory clues.

GPS navigation is commonplace in everyday life. While it has the capacity to make our lives easier, it is often used to automate functions that were once exclusively performed by our brain. Staying mentally active is key to healthy brain aging. Therefore, is GPS navigation causing more harm than good? Here we demonstrate that traditional turn-by-turn navigation promotes passive spatial navigation and ultimately, poor spatial learning of the surrounding environment. We propose an alternative form of GPS navigation based on sensory augmentation, that has the potential to fundamentally alter the way we navigate with GPS. By implementing a 3D spatial audio system similar to an auditory compass, users are directed towards their destination without explicit directions. Rather than being led passively through verbal directions, users are encouraged to take an active role in their own spatial navigation, leading to more accurate cognitive maps of space. Technology will always play a significant role in everyday life; however, it is important that we actively engage with the world around us. By simply rethinking the way we interact with GPS navigation, we can engage users in their own spatial navigation, leading to a better spatial understanding of the explored environment.

Virtual Body Ownership Illusions for Mental Health: A Narrative Review.

Over the last 20 years, virtual reality (VR) has been widely used to promote mental health in populations presenting different clinical conditions. Mental health does not refer only to the absence of psychiatric disorders but to the absence of a wide range of clinical conditions that influence people's general and social well-being such as chronic pain, neurological disorders that lead to motor o perceptual impairments, psychological disorders that alter behaviour and social cognition, or physical conditions like eating disorders or present in amputees. It is known that an accurate perception of oneself and of the surrounding environment are both key elements to enjoy mental health and well-being, and that both can be distorted in patients suffering from the clinical conditions mentioned above. In the past few years, multiple studies have shown the effectiveness of VR to modulate such perceptual distortions of oneself and of the surrounding environment through virtual body ownership illusions. This narrative review aims to review clinical studies that have explored the manipulation of embodied virtual bodies in VR for improving mental health, and to discuss the current state of the art and the challenges for future research in the context of clinical care.

A whole body characterization of individual strategies, gender differences, and common styles in overarm throwing.

Overarm throwing is a fundamental human skill. Since paleolithic hunter-gatherer societies, the ability of throwing played a key role in brain and body co-evolution. For decades, throwing skill acquisition has been the subject of developmental and gender studies. However, due to its complex multijoint nature, whole body throwing has found little space in quantitative studies of motor behavior. In this study we examined how overarm throwing varies within and between individuals in a sample of untrained adults. To quantitatively compare whole body kinematics across throwing actions, we introduced a new combination of spatiotemporal principal component, linear discrimination, and clustering analyses. We found that the identity and gender of a thrower can be robustly inferred by the kinematics of a single throw, reflecting the characteristic features in individual throwing strategies and providing a quantitative ground for the well-known differences between males and females in throwing behavior. We also identified four main classes of throwing strategies, stable within individuals and resembling the main stages of throwing proficiency acquisition during motor development. These results support earlier proposals linking interindividual and gender differences in throwing, with skill acquisition interrupted at different stages of the typical developmental trajectory of throwing motor behavior.NEW & NOTEWORTHY Unconstrained throwing, because of its complexity, received little attention in quantitative motor control studies. By introducing a new approach to analyze whole body kinematics, we quantitatively characterized gender effects, interindividual differences, and common patterns in nontrained throwers. The four throwing styles identified across individuals resemble different stages in the acquisition of throwing skills during development. These results advance our understanding of complex motor skills, bridging the gap between motor control, motor development, and sport science.

Where Are You Throwing the Ball? I Better Watch Your Body, Not Just Your Arm!

The ability to intercept or avoid a moving object, whether to catch a ball, snatch one's prey, or avoid the path of a predator, is a skill that has been acquired throughout evolution by many species in the animal kingdom. This requires processing early visual cues in order to program anticipatory motor responses tuned to the forthcoming event. Here, we explore the nature of the early kinematics cues that could inform an observer about the future direction of a ball projected with an unconstrained overarm throw. Our goal was to pinpoint the body segments that, throughout the temporal course of the throwing action, could provide key cues for accurately predicting the side of the outgoing ball. We recorded whole-body kinematics from twenty non-expert participants performing unconstrained overarm throws at four different targets placed on a vertical plane at 6 m distance. In order to characterize the spatiotemporal structure of the information embedded in the kinematics of the throwing action about the outgoing ball direction, we introduced a novel combination of dimensionality reduction and machine learning techniques. The recorded kinematics clearly shows that throwing styles differed considerably across individuals, with corresponding inter-individual differences in the spatio-temporal structure of the thrower predictability. We found that for most participants it is possible to predict the region where the ball hit the target plane, with an accuracy above 80%, as early as 400-500 ms before ball release. Interestingly, the body parts that provided the most informative cues about the action outcome varied with the throwing style and during the time course of the throwing action. Not surprisingly, at the very end of the action, the throwing arm is the most informative body segment. However, cues allowing for predictions to be made earlier than 200 ms before release are typically associated to other body parts, such as the lower limbs and the contralateral arm. These findings are discussed in the context of the sport-science literature on throwing and catching interactive tasks, as well as from the wider perspective of the role of sensorimotor coupling in interpersonal social interactions.

Intercepting virtual balls approaching under different gravity conditions: evidence for spatial prediction.

To accurately time motor responses when intercepting falling balls we rely on an internal model of gravity. However, whether and how such a model is also used to estimate the spatial location of interception is still an open question. Here we addressed this issue by asking 25 participants to intercept balls projected from a fixed location 6 m in front of them and approaching along trajectories with different arrival locations, flight durations, and gravity accelerations (0g and 1g). The trajectories were displayed in an immersive virtual reality system with a wide field of view. Participants intercepted approaching balls with a racket, and they were free to choose the time and place of interception. We found that participants often achieved a better performance with 1g than 0g balls. Moreover, the interception points were distributed along the direction of a 1g path for both 1g and 0g balls. In the latter case, interceptions tended to cluster on the upper half of the racket, indicating that participants aimed at a lower position than the actual 0g path. These results suggest that an internal model of gravity was probably used in predicting the interception locations. However, we found that the difference in performance between 1g and 0g balls was modulated by flight duration, the difference being larger for faster balls. In addition, the number of peaks in the hand speed profiles increased with flight duration, suggesting that visual information was used to adjust the motor response, correcting the prediction to some extent.NEW & NOTEWORTHY Here we show that an internal model of gravity plays a key role in predicting where to intercept a fast-moving target. Participants also assumed an accelerated motion when intercepting balls approaching in a virtual environment at constant velocity. We also show that the role of visual information in guiding interceptive movement increases when more time is available.

Concurrent talking in immersive virtual reality: on the dominance of visual speech cues.

Humans are good at selectively listening to specific target conversations, even in the presence of multiple concurrent speakers. In our research, we study how auditory-visual cues modulate this selective listening. We do so by using immersive Virtual Reality technologies with spatialized audio. Exposing 32 participants to an Information Masking Task with concurrent speakers, we find significantly more errors in the decision-making processes triggered by asynchronous audiovisual speech cues. More precisely, the results show that lips on the Target speaker matched to a secondary (Mask) speaker's audio severely increase the participants' comprehension error rates. In a control experiment (n = 20), we further explore the influences of the visual modality over auditory selective attention. The results show a dominance of visual-speech cues, which effectively turn the Mask into the Target and vice-versa. These results reveal a disruption of selective attention that is triggered by bottom-up multisensory integration. The findings are framed in the sensory perception and cognitive neuroscience theories. The VR setup is validated by replicating previous results in this literature in a supplementary experiment.

The sense of body ownership relaxes temporal constraints for multisensory integration.

Experimental work on body ownership illusions showed how simple multisensory manipulation can generate the illusory experience of an artificial limb as being part of the own-body. This work highlighted how own-body perception relies on a plastic brain representation emerging from multisensory integration. The flexibility of this representation is reflected in the short-term modulations of physiological states and perceptual processing observed during these illusions. Here, we explore the impact of ownership illusions on the temporal dimension of multisensory integration. We show that, during the illusion, the temporal window for integrating touch on the physical body with touch seen on a virtual body representation, increases with respect to integration with visual events seen close but separated from the virtual body. We show that this effect is mediated by the ownership illusion. Crucially, the temporal window for visuotactile integration was positively correlated with participants' scores rating the illusory experience of owning the virtual body and touching the object seen in contact with it. Our results corroborate the recently proposed causal inference mechanism for illusory body ownership. As a novelty, they show that the ensuing illusory causal binding between stimuli from the real and fake body relaxes constraints for the integration of bodily signals.

The body fades away: investigating the effects of transparency of an embodied virtual body on pain threshold and body ownership.

The feeling of "ownership" over an external dummy/virtual body (or body part) has been proven to have both physiological and behavioural consequences. For instance, the vision of an "embodied" dummy or virtual body can modulate pain perception. However, the impact of partial or total invisibility of the body on physiology and behaviour has been hardly explored since it presents obvious difficulties in the real world. In this study we explored how body transparency affects both body ownership and pain threshold. By means of virtual reality, we presented healthy participants with a virtual co-located body with four different levels of transparency, while participants were tested for pain threshold by increasing ramps of heat stimulation. We found that the strength of the body ownership illusion decreases when the body gets more transparent. Nevertheless, in the conditions where the body was semi-transparent, higher levels of ownership over a see-through body resulted in an increased pain sensitivity. Virtual body ownership can be used for the development of pain management interventions. However, we demonstrate that providing invisibility of the body does not increase pain threshold. Therefore, body transparency is not a good strategy to decrease pain in clinical contexts, yet this remains to be tested.

Allocentric and egocentric manipulations of the sense of self-location in full-body illusions and their relation with the sense of body ownership.

Self-location refers to the experience of occupying a given position in the environment. Recent research has addressed the sense of self-location as one of the key components of self-consciousness, together with the experience of owning the physical body (ownership) (Blanke and Metzinger, Trends Cogn Sci 13:7-13 in 2009. doi: 10.1016/j.tics.2008.10.003 ). Experimentally controlled full-body illusions proved to be valuable research tools to study these components and their interaction, and to explore their underlying neural underpinning. The focus of this manuscript is to provide a close look into the nuances of different illusory experiences affecting the sense of self-location and to examine their relation to the concurrent experienced sense of body ownership. On the basis of previous reviewed studies, it is proposed that the sense of self-location may be regarded as the blending of two paralllel representations: the abstract allocentric coding of the position occupied in the environment, mainly associated with visual-perspective, and the egocentric mapping of somatosensory sensations into the external space, mainly associated with peripersonal space. Open questions to be addressed by future research are further addressed.

Over my fake body: body ownership illusions for studying the multisensory basis of own-body perception.

Which is my body and how do I distinguish it from the bodies of others, or from objects in the surrounding environment? The perception of our own body and more particularly our sense of body ownership is taken for granted. Nevertheless, experimental findings from body ownership illusions (BOIs), show that under specific multisensory conditions, we can experience artificial body parts or fake bodies as our own body parts or body, respectively. The aim of the present paper is to discuss how and why BOIs are induced. We review several experimental findings concerning the spatial, temporal, and semantic principles of crossmodal stimuli that have been applied to induce BOIs. On the basis of these principles, we discuss theoretical approaches concerning the underlying mechanism of BOIs. We propose a conceptualization based on Bayesian causal inference for addressing how our nervous system could infer whether an object belongs to our own body, using multisensory, sensorimotor, and semantic information, and we discuss how this can account for several experimental findings. Finally, we point to neural network models as an implementational framework within which the computational problem behind BOIs could be addressed in the future.

Sliding perspectives: dissociating ownership from self-location during full body illusions in virtual reality.

Bodily illusions have been used to study bodily self-consciousness and disentangle its various components, among other the sense of ownership and self-location. Congruent multimodal correlations between the real body and a fake humanoid body can in fact trigger the illusion that the fake body is one's own and/or disrupt the unity between the perceived self-location and the position of the physical body. However, the extent to which changes in self-location entail changes in ownership is still matter of debate. Here we address this problem with the support of immersive virtual reality. Congruent visuotactile stimulation was delivered on healthy participants to trigger full body illusions from different visual perspectives, each resulting in a different degree of overlap between real and virtual body. Changes in ownership and self-location were measured with novel self-posture assessment tasks and with an adapted version of the cross-modal congruency task. We found that, despite their strong coupling, self-location and ownership can be selectively altered: self-location was affected when having a third person perspective over the virtual body, while ownership toward the virtual body was experienced only in the conditions with total or partial overlap. Thus, when the virtual body is seen in the far extra-personal space, changes in self-location were not coupled with changes in ownership. If a partial spatial overlap is present, ownership was instead typically experienced with a boosted change in the perceived self-location. We discussed results in the context of the current knowledge of the multisensory integration mechanisms contributing to self-body perception. We argue that changes in the perceived self-location are associated to the dynamical representation of peripersonal space encoded by visuotactile neurons. On the other hand, our results speak in favor of visuo-proprioceptive neuronal populations being a driving trigger in full body ownership illusions.